{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cardea\n",
    "\n",
    "<img src=\"img/cardea.png\" style=\"width: 200px;\">\n",
    "\n",
    "Cardea is a machine learning library built on top of [FHIR](https://www.hl7.org/fhir/) data standard to solve various common prediction problems from electronic health records. \n",
    "\n",
    "This is a python notebook that demonstrates Cardea's workflow from a user's perspective. It is decomposed based on the elements present in the framework. Documentation: https://d3-ai.github.io/Cardea/\n",
    "\n",
    "Currently in support of **version 0.0.2**.\n",
    "\n",
    "In this tutorial, we show how to predict whether a patient will showup to an appointment using a dataset from Kaggle's [Medical Appointment No Shows](https://www.kaggle.com/joniarroba/noshowappointments). Over 30% of patients miss their scheduled appointments, this results in poor optimization of time and resources. Through machine learning, we want to predict future appointment no-shows by using an end-to-end library that is easy to interpret."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "paramiko missing, opening SSH/SCP/SFTP paths will be disabled.  `pip install paramiko` to suppress\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from cardea import Cardea"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# optional\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cardea\n",
    "\n",
    "After importing the necessary packages, it is time to initialize a new object of cardea. This object will serve as the main pillar to call any method within cardea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd = Cardea()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Kaggle Dataset \n",
    "\n",
    "Using cardea's `load_data_entityset`, we can now either load local files that are in [FHIR](https://hl7.org/fhir) format. In order to try out cardea, we want to load kaggle's open dataset instead. Cardea automatically loads the Kaggle dataset into its memory when no folder path is given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd.load_data_entityset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: fhir\n",
       "  Entities:\n",
       "    Address [Rows: 81, Columns: 2]\n",
       "    Appointment_Participant [Rows: 6100, Columns: 2]\n",
       "    Appointment [Rows: 110527, Columns: 5]\n",
       "    CodeableConcept [Rows: 4, Columns: 2]\n",
       "    Coding [Rows: 3, Columns: 2]\n",
       "    Identifier [Rows: 227151, Columns: 1]\n",
       "    Observation [Rows: 110527, Columns: 3]\n",
       "    Patient [Rows: 6100, Columns: 4]\n",
       "    Reference [Rows: 6100, Columns: 1]\n",
       "  Relationships:\n",
       "    Appointment_Participant.actor -> Reference.identifier\n",
       "    Appointment.participant -> Appointment_Participant.object_id\n",
       "    CodeableConcept.coding -> Coding.object_id\n",
       "    Observation.code -> CodeableConcept.object_id\n",
       "    Observation.subject -> Reference.identifier\n",
       "    Patient.address -> Address.object_id"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to view the loaded entityset\n",
    "cd.es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first section (entities) represents the resources that were loaded into the framework. In other words, it describes the dataframes available presented with the number of rows and columns. The second section describes the relationship between the resources. For example, the `Patient` resource has an address variable that is connected to the `Address` resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Definition\n",
    "\n",
    "You can display all the problems currently implemented in cardea under the `list_problems` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'DiagnosisPrediction',\n",
       " 'LengthOfStay',\n",
       " 'MissedAppointmentProblemDefinition',\n",
       " 'MortalityPrediction',\n",
       " 'ProlongedLengthOfStay',\n",
       " 'Readmission'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cd.list_problems()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we will define the problem as ‘Missed Appointment’ to predict whether a patient will miss their next appointment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select problem\n",
    "cutoff = cd.select_problem('MissedAppointmentProblemDefinition')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AutoML\n",
    "Automated machine learning composes from two main phases:\n",
    "1. automated feature engineering: through autofe, we extract information called features. Finding the features is crucial for building data models and help in finding a satisfactory answer and interpreting the dataset as a whole.\n",
    "2. automated modeling: in automated modeling, the library supports running multiple machine learning algorithms and optimizes its hyperparamters in order to find the most optimal model.\n",
    "\n",
    "Typically, this phase is complex and comprises of many elements, but **Cardea** provides an easier way of handling both phases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 13 features\n",
      "Elapsed: 00:41 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 10/10 chunks\n"
     ]
    }
   ],
   "source": [
    "# feature engineering\n",
    "feature_matrix = cd.generate_features(cutoff[:1000]) # first 1000 records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffle the dataframe\n",
    "feature_matrix = feature_matrix.sample(frac=1)\n",
    "\n",
    "# pop the target labels\n",
    "y = list(feature_matrix.pop('label'))\n",
    "X = feature_matrix.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pipeline variable represents the order in which machine learning algorithms are executed. It can be used to compare models together by specifying multiple algorithms in different lists. Such as:\n",
    "```\n",
    "pipeline = [['sklearn.ensemble.RandomForestClassifier'], \n",
    "            ['sklearn.neighbors.KNeighborsClassifier']]\n",
    "```\n",
    "Here we execute two different models, the first one being Random Forest and the second is K-Nearest Neighbor (KNN).\n",
    "In addition, you can use the pipeline to create your own encoding and modeling pipeline where the data crosses several algorithms to create the prediction model. Such as:\n",
    "```\n",
    "pipeline = [['sklearn.preprocessing.StandardScaler', 'sklearn.ensemble.RandomForestClassifier'], \n",
    "            ['sklearn.neighbors.KNeighborsClassifier']]\n",
    "```\n",
    "Here there are two different models, the first one composes of two primitives (Preprocessing through normalization then applying Random Forest) and the second is basic KNN.\n",
    "More on machine learning algorithms and MLPrimitives can be found here: https://HDI-Project.github.io/MLPrimitives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# modeling\n",
    "pipeline = [['sklearn.ensemble.RandomForestClassifier'], \n",
    "            ['sklearn.naive_bayes.MultinomialNB'], \n",
    "            ['sklearn.neighbors.KNeighborsClassifier']]\n",
    "\n",
    "exe = cd.execute_model(feature_matrix=X,\n",
    "                       target=y, \n",
    "                       primitives=pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After executing the pipelines, the method returns a list composing of each pipeline with each fold representing three main results:\n",
    "* The list of primitives used.\n",
    "* The actual label vector.\n",
    "* The predicted label vector.\n",
    "* The tuned hyperparameters (if given).\n",
    "\n",
    "In order to perceive the results and look at the performance of each pipeline we can view it's training process by merely plotting the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbsAAAGrCAYAAABZvzmFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmUZWV97vHvE1rQONHQigQQMcEBo4J2MM6oKEMScUUTcQSvLKKRZF1NbsRrFhrUXPQOerOuEzEIagSNcehEDIJKTIIobcIgyNC0iXQDgt2IM4j+7h/7LbK7uqqruutQ3fX6/ax1Vu39vu/e53d2nTrP2fvssytVhSRJPfuF7V2AJEl3NcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTosqyT2S/F2SW5P8zQLW86Ikn51kbdtDks8kOWYbl71fkiuT3GPSdWl2SXZp2/1+27sWzZ9hpxkleWGS1Um+n+SG9qL8pAms+nnAHsDuVfU727qSqvrrqnrWBOrZRJJDklSST0xrf3RrP3+e63ljkg/NNa6qjqiqM7ax3BOB06vqR9u4fHeSvDrJjUm+m+S0JLtsYewvJnlXkm+3N19fnLaetW091yd5e5JlAFV1G3Aaw/bXEmHYaTNJXgO8A/hzhmB6IPAu4KgJrH5f4OqqumMC67qr3Aw8Psnuo7ZjgKsndQcZbPPfX3sRPwaYM1AXS5KdtvP9H8YQQM9geJ49GPizLSxyKrAb8PD289WjvlXAY6rqPsCvAo8G/nDU/2HgmC2FqXYwVeXN25034L7A94Hf2cKYXRjC8Pp2ewewS+s7BFgH/BFwE3AD8LLW92fA7cBP2n28HHgj8KHRuh8EFLCszR8LrAW+B3wDeNGo/Z9Hyz0BuAi4tf18wqjvfOBNwL+09XwWWDHLY5uq/z3Aq1rbTsB64CTg/NHY/wtcB3wX+Crw5NZ++LTHecmojre0On4E/EprO671vxv429H63wp8DsgMdT4FWDOt7WXA19tjXAv83rT+o4CLW73XAoe39t2A97ff5S3AJ2faxq2tgF9p06e3ms8GfgAcCvwG8G/tPq4D3jht+ScBFwDfaf3HAr8GfAvYaTTut6e221Y8dz8M/Plo/hnAjbOMfVir8T7zWO/uwHnAu6a1XwM8dXv/zXqb5/Njexfgbce6tRfqO2hhM8uYk4ELgfsD92svXm9qfYe05U8G7gYcCfwQWN7638im4TZ9/kHtBXUZcM/2gvTQ1rcn8Ig2fecLcXuxvgV4SVvuBW1+99Z/fntxfwhwjzZ/yiyP7RCGsHsC8OXWdiRwDnAcm4bdi9sL4TKGcL8RuPtMj2tUxzeBR7Rl7samYfeLDHuPxwJPBr4N7D1Lna8CPj2t7TeAXwYCPLVt98e0voMZ3gg8k+GIzl7Aw1rfp4GPAMtbTU+dvo1H9zE97G4FntjWefe2/R7Z5h/FEGLPaeP3ZQjiF7T72R04sPVdARwxup9PAH/Upl/IEI6z3R7Yxl0CPH+0jhWt3t1n2H4vBS4D3t6282XAc6eNeSHD868Y9vYfPa1/FfCH2/tv1tv8bh7G1HS7A9+uLR9mfBFwclXdVFU3M+yxvWTU/5PW/5OqOpth7+ah21jPz4BfTXKPqrqhqi6fYcxvANdU1Qer6o6qOhO4Evit0Zj3V9XVNXy+9VHgwC3daVVdAOyW5KEML4wfmGHMh6pqQ7vP/82wxzvX4zy9qi5vy/xk2vp+yLAd/w/D4ck/qKp1s6xnV4bgGC//6aq6tgb/yLAH++TW/XLgtKo6t6p+VlXrq+rKJHsCRwCvqKpb2u/sH+d4DGOfqqp/aev8cVWdX1WXtflLgTMZgheG8Divqs5s97Ohqi5ufWcwvHkgyW7AYQx7alTVh6tq1y3cvtnWcS+G8J0yNX3vGerem+Hw5K3ALwEnAGckefhoe364hsOYD2HY0//WtHV8j+H3oCXAsNN0G4AVUx/Gz+KXgP8Yzf9Ha7tzHdPC8ocML0Rbpap+ADwfeAVwQ5JPJ3nYPOqZqmmv0fyN21DPBxleBJ/GsKexiSR/nOTr7eSG7zAcAl4xxzqv21JnVX2Z4RBkGEJ5Nrcw7UU8yRFJLkyysdVz5KiefRj2bqfbB9hYVbfMUfdsNnk8SR6X5AtJbk5yK8Pvbq4aYAj330pyT+B3gX+qqhu2spbvA/cZzU9Nf2+GsT9ieFP25qq6vQX8F4DNTnqqqmuAyxk+tx67N8OepZYAw07TfQm4DXjOFsZcz3BIasoDW9u2+AHD4bspDxh3VtU5VfVMhkOYVwJ/OY96pmpav401Tfkg8PvA2W2v605Jngz8CcML8/Kq2pVhLyFTpc+yzi3+m5Ekr2LYQ7y+rX82lzLscUwttwvwt8D/AvZo9Zw9quc6hkOc013HsAc70x7KJr+bJA+YYcz0x/NhhsN7+1TVfRn2iOaqgapaz/Dc+22GvdsPju73Re2s4NluD2xDL2c4kWTKo4FvVdWGGe7y0nk8lrFlM9T+cIZDp1oCDDttoqpuZTgR451JntNOz75b22t4Wxt2JvCn7XteK9r4bT0r8GLgKUkemOS+wOumOpLskeSo9m7/NoZ37j+bYR1nAw9pX5dYluT5wAHA329jTQBU1TcYDsG9fobuezN8NnkzsCzJSWy6V/Et4EFbc8ZlkocAb2Y4nPcS4E+SzHa49SvArkmm9l53ZgjJm4E7khzBpnspfwW8LMkzkvxCkr2SPKztPX0GeFeS5e13/ZS2zCXAI5IcmOTuDJ9DzuXeDHuKP05yMMOhyyl/DRya5Hfb72n3aY/vAwwB/0jg41ONNXzN5F5buH1ztPzLkxzQwvtPGT5XnMkXGT4/fV2r5YkMe/DnACQ5Lsn92/QBDM/Lz00t3Lb7bgyfXWsJMOy0mfb502sYXixuZnhHfgLwyTbkzcBqhnfHlwH/2tq25b7OZTg54lKGMxrHAfULrY7rgY0MwfPKGdaxAfhNhpNENjC8YP5mVX17W2qatu5/rqqZ9lrPAf6B4YSS/wB+zKaH9Ka+ML8hyb/OdT/tsPGHgLdW1SXt0Nl/Bz440+ntVXU7wwv5i9v89xhOjf8owyHOFzLsYU2N/wrD2ZpvZ9gD/Uf+c2/4JQyH9K5kOIP2v7ZlrmY40eg8hjMP/3mux8GwJ3xyku8xvAm681BsC6UjGX5PGxne6Iz3xD7RavrE9D3p+aiqfwDexnA48psMv5c3TPUnuTzJi9rYnzCcnXokw/b4S+ClVXVlG/5E4LIkP2B4M3U2w+9jyguBM2r4zp2WgFT5z1ulpSjDFTz+CTioOvlieZJrGb4ycd72rmU27c3HJcBTquqm7V2P5sewk7RDSPJchu8WPqSqZjpcLW2zBR3GTLJbknOTXNN+Lp9l3E+TXNxuq0bt+yX5cpI1ST6SZOeF1CNpacpwGbZ3M3yR36DTxC1oz66dsLCxqk5JciLDWWmvnWHc96tqs1O9k3wU+HhVnZXkPQxXTHj3NhckSdIMFhp2VwGHVNUN7cup51fVZl+qnSnskoTh5IcHVNUdSR7PcGmhw7a5IEmSZrClLw7Pxx6jL37eyHDR4JncPclqhlO1T6mqTzJcqeM7oy8fr2PTLwFvIsnxwPEA97znPR/70IfO9N1iqV8/uG1Hvna2dNe4cf11fOeWDZl75JbNGXZJzmPaF32bTb57VFWVZLbdxH2ran2SBwOfT3IZm17WZ05VdSrDVcp57GNX1r98efXWLC4teV9Zu3F7lyAtuuN/++kTWc+cYVdVh87Wl+RbSfYcHcac8TTcdnUEqmpt+yD6IIarPeyaZFnbu9ubhV/xQpKkzSz0S+WrGP6nFu3np6YPaFdl2KVNr2D4suYVNXxY+AWGf+Y56/KSJC3UQsPuFOCZSa5h+F9WpwAkWZnkfW3Mw4HVSS5hCLdTquqK1vda4DVJ1jB8hvdXC6xHkqTNLOgElXaZpmfM0L6a4X9/Tf2rlEfOsvxahv+zJUnSXcZrY0qSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrq3oLBLsluSc5Nc034un2HMgUm+lOTyJJcmef6o7/Qk30hycbsduJB6JEmayUL37E4EPldV+wOfa/PT/RB4aVU9AjgceEeSXUf9/62qDmy3ixdYjyRJm1lo2B0FnNGmzwCeM31AVV1dVde06euBm4D7LfB+JUmat4WG3R5VdUObvhHYY0uDkxwM7AxcO2p+Szu8+fYku2xh2eOTrE6y+uZv37zAsiVJP0/mDLsk5yX52gy3o8bjqqqA2sJ69gQ+CLysqn7Wml8HPAz4NWA34LWzLV9Vp1bVyqpaeb8V7hhKkuZv2VwDqurQ2fqSfCvJnlV1Qwuzm2YZdx/g08Drq+rC0bqn9gpvS/J+4I+3qnpJkuZhoYcxVwHHtOljgE9NH5BkZ+ATwAeq6mPT+vZsP8Pwed/XFliPJEmbWWjYnQI8M8k1wKFtniQrk7yvjfld4CnAsTN8xeCvk1wGXAasAN68wHokSdrMnIcxt6SqNgDPmKF9NXBcm/4Q8KFZln/6Qu5fkqT58AoqkqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO5NJOySHJ7kqiRrkpw4Q/8uST7S+r+c5EGjvte19quSHDaJeiRJGltw2CXZCXgncARwAPCCJAdMG/Zy4Jaq+hXg7cBb27IHAEcDjwAOB97V1idJ0sRMYs/uYGBNVa2tqtuBs4Cjpo05CjijTX8MeEaStPazquq2qvoGsKatT5KkiZlE2O0FXDeaX9faZhxTVXcAtwK7z3NZAJIcn2R1ktU3f/vmCZQtSfp5sWROUKmqU6tqZVWtvN+K+23vciRJS8gkwm49sM9ofu/WNuOYJMuA+wIb5rmsJEkLMomwuwjYP8l+SXZmOOFk1bQxq4Bj2vTzgM9XVbX2o9vZmvsB+wNfmUBNkiTdadlCV1BVdyQ5ATgH2Ak4raouT3IysLqqVgF/BXwwyRpgI0Mg0sZ9FLgCuAN4VVX9dKE1SZI0tuCwA6iqs4Gzp7WdNJr+MfA7syz7FuAtk6hDkqSZLJkTVCRJ2laGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7Ewm7JIcnuSrJmiQnztD/miRXJLk0yeeS7Dvq+2mSi9tt1STqkSRpbNlCV5BkJ+CdwDOBdcBFSVZV1RWjYf8GrKyqHyZ5JfA24Pmt70dVdeBC65AkaTaT2LM7GFhTVWur6nbgLOCo8YCq+kJV/bDNXgjsPYH7lSRpXiYRdnsB143m17W22bwc+Mxo/u5JVie5MMlzZlsoyfFt3Oqbv33zwiqWJP1cWfBhzK2R5MXASuCpo+Z9q2p9kgcDn09yWVVdO33ZqjoVOBXgsY9dWYtSsCSpC5PYs1sP7DOa37u1bSLJocDrgWdX1W1T7VW1vv1cC5wPHDSBmiRJutMkwu4iYP8k+yXZGTga2OSsyiQHAe9lCLqbRu3Lk+zSplcATwTGJ7ZIkrRgCz6MWVV3JDkBOAfYCTitqi5PcjKwuqpWAf8TuBfwN0kAvllVzwYeDrw3yc8YgveUaWdxSpK0YBP5zK6qzgbOntZ20mj60FmWuwB45CRqkCRpNl5BRZLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktS9iYRdksOTXJVkTZITZ+g/NsnNSS5ut+NGfcckuabdjplEPZIkjS1b6AqS7AS8E3gmsA64KMmqqrpi2tCPVNUJ05bdDXgDsBIo4Ktt2VsWWpckSVMmsWd3MLCmqtZW1e3AWcBR81z2MODcqtrYAu5c4PAJ1CRJ0p0WvGcH7AVcN5pfBzxuhnHPTfIU4Grg1VV13SzL7jXXHf7oJz/l6+u/u+0VS0vQYc8/aXuXIC2629ZeP5H1LNYJKn8HPKiqHsWw93bG1q4gyfFJVidZfcvGDRMvUJLUr0mE3Xpgn9H83q3tTlW1oapua7PvAx4732VH6zi1qlZW1crlu+0+gbIlST8vJhF2FwH7J9kvyc7A0cCq8YAke45mnw18vU2fAzwryfIky4FntTZJkiZmwZ/ZVdUdSU5gCKmdgNOq6vIkJwOrq2oV8IdJng3cAWwEjm3LbkzyJobABDi5qjYutCZJksYmcYIKVXU2cPa0tpNG068DXjfLsqcBp02iDkmSZuIVVCRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd0z7CRJ3TPsJEndM+wkSd2bSNglOTzJVUnWJDlxhv63J7m43a5O8p1R309HfasmUY8kSWPLFrqCJDsB7wSeCawDLkqyqqqumBpTVa8ejf8D4KDRKn5UVQcutA5JkmYziT27g4E1VbW2qm4HzgKO2sL4FwBnTuB+JUmal0mE3V7AdaP5da1tM0n2BfYDPj9qvnuS1UkuTPKc2e4kyfFt3OpbNm6YQNmSpJ8Xi32CytHAx6rqp6O2fatqJfBC4B1JfnmmBavq1KpaWVUrl++2+2LUKknqxCTCbj2wz2h+79Y2k6OZdgizqta3n2uB89n08zxJkhZsEmF3EbB/kv2S7MwQaJudVZnkYcBy4EujtuVJdmnTK4AnAldMX1aSpIVY8NmYVXVHkhOAc4CdgNOq6vIkJwOrq2oq+I4GzqqqGi3+cOC9SX7GELynjM/ilCRpEhYcdgBVdTZw9rS2k6bNv3GG5S4AHjmJGiRJmo1XUJEkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHVvImGX5LQkNyX52iz9SfIXSdYkuTTJY0Z9xyS5pt2OmUQ9kiSNTWrP7nTg8C30HwHs327HA+8GSLIb8AbgccDBwBuSLJ9QTZIkARMKu6r6IrBxC0OOAj5QgwuBXZPsCRwGnFtVG6vqFuBcthyakiRttcX6zG4v4LrR/LrWNlv7ZpIcn2R1ktW3bNxwlxUqSerPkjlBpapOraqVVbVy+W67b+9yJElLyGKF3Xpgn9H83q1ttnZJkiZmscJuFfDSdlbmrwO3VtUNwDnAs5IsbyemPKu1SZI0McsmsZIkZwKHACuSrGM4w/JuAFX1HuBs4EhgDfBD4GWtb2OSNwEXtVWdXFVbOtFFkqStNpGwq6oXzNFfwKtm6TsNOG0SdUiSNJMlc4KKJEnbyrCTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHVvImGX5LQkNyX52iz9L0pyaZLLklyQ5NGjvn9v7RcnWT2JeiRJGpvUnt3pwOFb6P8G8NSqeiTwJuDUaf1Pq6oDq2rlhOqRJOlOyyaxkqr6YpIHbaH/gtHshcDek7hfSZLmY3t8Zvdy4DOj+QI+m+SrSY6fbaEkxydZnWT1LRs33OVFSpL6MZE9u/lK8jSGsHvSqPlJVbU+yf2Bc5NcWVVfnL5sVZ1KO/x5wKMOqkUpWJLUhUXbs0vyKOB9wFFVdeeuWVWtbz9vAj4BHLxYNUmSfj4sStgleSDwceAlVXX1qP2eSe49NQ08C5jxjE5JkrbVRA5jJjkTOARYkWQd8AbgbgBV9R7gJGB34F1JAO5oZ17uAXyitS0DPlxV/zCJmiRJmjKpszFfMEf/ccBxM7SvBR69+RKSJE2OV1CRJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1byJhl+S0JDcl+dos/YckuTXJxe120qjv8CRXJVmT5MRJ1CNJ0tik9uxOBw6fY8w/VdWB7XYyQJKdgHcCRwAHAC9IcsCEapIkCZhQ2FXVF4GN27DowcCaqlpbVbcDZwFHTaImSZKmLFvE+3p8kkuA64E/rqrLgb2A60Zj1gGPm2nhJMcDx7fZ2x7zoPvOeMh0CVgBfHt7F7EA1r/9LOXaYWnXv5Rrh6Vd/0MnsZLFCrt/Bfatqu8nORL4JLD/1qygqk4FTgVIsrqqVk6+zLveUq4drH97Wsq1w9KufynXDku7/iSrJ7GeRTkbs6q+W1Xfb9NnA3dLsgJYD+wzGrp3a5MkaWIWJeySPCBJ2vTB7X43ABcB+yfZL8nOwNHAqsWoSZL082MihzGTnAkcAqxIsg54A3A3gKp6D/A84JVJ7gB+BBxdVQXckeQE4BxgJ+C09lneXE6dRN3byVKuHax/e1rKtcPSrn8p1w5Lu/6J1J4hcyRJ6pdXUJEkdc+wkyR1b4cNuyS7JTk3yTXt5/JZxv10dBmyVaP2/ZJ8uV2G7CPtBJgdpvYkByb5UpLLk1ya5PmjvtOTfGP0uA5cpLq3eOm2JLu0bbmmbdsHjfpe19qvSnLYYtQ7rba5an9Nkivatv5ckn1HfTM+hxbTPOo/NsnNozqPG/Ud055r1yQ5ZnErn1ftbx/VfXWS74z6tuu2n8elDpPkL9pjuzTJY0Z923W7txrmqv9Fre7LklyQ5NGjvn9v7RdP6vT+rTGP2id7mcmq2iFvwNuAE9v0icBbZxn3/VnaP8pwIgzAe4BX7ki1Aw8B9m/TvwTcAOza5k8HnrfI23sn4FrgwcDOwCXAAdPG/D7wnjZ9NPCRNn1AG78LsF9bz047WO1PA36xTb9yqvYtPYd2sPqPBf7fDMvuBqxtP5e36eU7Uu3Txv8Bw4loO8q2fwrwGOBrs/QfCXwGCPDrwJd3hO2+FfU/YaouhssyfnnU9+/Aih142x8C/P1Cn3NTtx12z47hsmFntOkzgOfMd8EkAZ4OfGxblp+AOWuvqqur6po2fT1wE3C/Ratwc/O5dNv4cX0MeEbb1kcBZ1XVbVX1DWBNW99imbP2qvpCVf2wzV7I8J3OHcVCLpt3GHBuVW2sqluAc5n7OrWTtLW1vwA4c1Eqm4ea+1KHRwEfqMGFwK5J9mT7b3dg7vqr6oJWH+xgz/t5bPvZbNPfy44cdntU1Q1t+kZgj1nG3T3J6iQXJpkKld2B71TVHW1+HcOlyRbLfGsH7vzu4c4M71amvKUdfnh7kl3uojrHZrp02/RtdueYtm1vZdjW81n2rrS19/9yhnfrU2Z6Di2m+db/3Pac+FiSqYsxLJlt3w4d7wd8ftS8vbf9XGZ7fNt7u2+L6c/7Aj6b5KsZLse4I3p8kkuSfCbJI1rbNm37xbw25maSnAc8YIau149nqqqSzPYdiX2ran2SBwOfT3IZw4vwXWpCtdPeJX4QOKaqftaaX8cQkjszfMfktcDJk6j7512SFwMrgaeOmjd7DlXVtTOvYbv5O+DMqrotye8x7GE/fTvXtLWOBj5WVT8dtS2Fbb/kJXkaQ9g9adT8pLbt7w+cm+TKtre1o1jwZSbHtuueXVUdWlW/OsPtU8C3WhBMBcJNs6xjffu5FjgfOIjh6iy7JpkK84lfhmwStSe5D/Bp4PXtEMnUum9oh01uA97P4hwSnM+l2+4c07btfRm29fa+7Nu87j/JoQxvRp7dti0w63NoMc1Zf1VtGNX8PuCx8132LrY193+ktxruAAAB+UlEQVQ00w5h7gDbfi6zPb7tvd3nLcmjGJ4zR1XVhqn20ba/CfgEi/vRw5xqwpeZ3JEPY64Cps5wOgb41PQBSZZPHeJrG+GJwBU1fIr5BYYrt8y6/F1oPrXvzPAE+0BVfWxa31RQhuHzvsX4Dw/zuXTb+HE9D/h829argKMznK25H8O7r68sQs1T5qw9yUHAexmC7qZR+4zPoUWrfDCf+vcczT4b+HqbPgd4Vnscy4FntbbFMq9L/iV5GMOJHF8ate0I234uq4CXtrMyfx24tX1Esb23+7wkeSDwceAlVXX1qP2eSe49Nc1Q/w71n2Qy6ctMbs3ZM4t5Y/gs6HPANcB5wG6tfSXwvvrPM40uYzgb5zLg5aPlH8zwgrsG+Btglx2s9hcDPwEuHt0ObH2fb4/na8CHgHstUt1HAlczfHb4+tZ2MkNAANy9bcs1bds+eLTs69tyVwFHbIfny1y1nwd8a7StV831HNrB6v8fwOWtzi8ADxst+1/a72QN8LIdrfY2/0bglGnLbfdtz7CneUP7W1zHcKjvFcArWn8Y/sH0ta3GlTvKdp9n/e8Dbhk971e39ge37X5Je169fges/YTRc/5C4Albes7NdfNyYZKk7u3IhzElSZoIw06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktS9/w9uKVFges0fHgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "fig = plt.figure(figsize=(7, 7))\n",
    "\n",
    "y_test = []\n",
    "y_pred = []\n",
    "for i in range(0, 10):\n",
    "    y_test.extend(exe['pipeline0']['folds'][str(i)]['Actual'])\n",
    "    y_pred.extend(exe['pipeline0']['folds'][str(i)]['predicted'])\n",
    "\n",
    "y_test = pd.Categorical(pd.Series(y_test)).codes\n",
    "y_pred = pd.Categorical(pd.Series(y_pred)).codes\n",
    "\n",
    "plt.title(\"Confusion Matrix (accuracy=%.2f)\" % accuracy_score(y_test, y_pred))\n",
    "plt.imshow(pd.crosstab(y_test, y_pred), cmap=\"Blues\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
